MULTI-ASPECTUAL KNOWLEDGE ELICITATION FOR 
PROCUREMENT OPTIMIZATION IN A WAREHOUSE COMPANY 


Franck Romuald Fotso Mtope, Sina Joneidy, Diptangshu Pandit & Farzad Rah 
School of Computing, Engineering & Digital Technologies, Teesside University, Middlesbrough, UK 


ABSTRACT: Efficient optimization of business processes required a profound understanding of expertise 
provided by domain specialists. However, extracting such insights can indeed be a laborious and time-consuming 
endeavour. This paper introduces the Multi-Aspectual Knowledge Elicitation framework (MAKE4ML) — a novel 
approach designed to effortlessly and effectively extract valuable information from domain experts. This 
framework inherently facilitates the development of machine-learning models capable of optimizing business 
processes, thereby diminishing reliance on experts. The framework's application within a food warehouse 
company is showcased, specifically targeting the enhancement of the procurement process. The employed 
methodology revolves around conducting comprehensive interviews with procurement experts, thereby enabling 
a meticulous exploration of diverse facets inherent to a business process. Subsequently, the gathered insights are 
employed to conceive and calibrate a machine learning model (time series forecasting). This model effectively 
emulates the domain experts’ proficiency, offering invaluable decision-oriented insights. The outcomes of this 
study show that our framework allows efficient knowledge elicitation, which is a pivotal factor in formulating and 
deploying a bespoke machine-learning model. The proposed approach can be extended into various other 
business processes, thereby paving the way for operational refinement, cost reduction, and amplified efficiency. 
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1. INTRODUCTION 


The growing demand for digitalization and process optimization has led to the integration of machine learning 
(ML) models across industries. This integration often requires insights from domain experts, necessitating the 
extraction of pertinent information to design tailored ML models. Researchers and ML engineers have employed 
various techniques, such as feature selection and knowledge elicitation, to enhance model accuracy while ensuring 
successful technology adoption. Studies have highlighted the challenges of knowledge elicitation, which 
significantly affect ML performance across disciplines. Researchers are increasingly exploring human 
involvement in ML workflows (D’Angelo & Palmieri, 2020; Park et al., 2023; Sundin et al., 2022; Wang et al., 
2021), combining expert knowledge with data from diverse sources (Ademujimi & Prabhu, 2021; Ben Brahim et 
al., 2022; Hu et al., 2019; Huang et al., 2019; Lee et al., 2020; Seymoens et al., 2019), and innovative ways to 
extract insights (Afrabandpey et al., 2019; Campos et al., 2018; Cheung et al., 2011; Crerie et al., 2009; El-Assady 
et al., 2020; El-Assady et al., 2019; Mantik et al., 2022; Mozina et al., 2018; Park et al., 2021; Yazici et al., 2022; 
Young et al., 2022). 


First, human involvement in ML workflows referred to as "Human-in-the-loop", aims to create cost-effective 
prediction models by incorporating human knowledge during data preparation and refinement stages. Secondly, 
the integration of expert-derived knowledge with data from sources like sensors refines training objectives and 
contextual alignment, as standard sensor data might lack external factors' consideration. Finally, Intuitive 
techniques (e.g., decision-mining, process mining) bridge gaps between ML engineers and multi-disciplinary 
experts, translating meaningful insights into ML model specifications. 


The central research question is: "How can we extract meaningful knowledge from domain experts for designing 
ML models while ensuring user acceptance?" To address this, a comprehensive framework is proposed, involving 
multi-aspectual knowledge extraction, translation into ML specifications, visualizing business workflows, and 
capturing decision-making rules and constraints. Key contributions include a multi-disciplinary knowledge 
extraction framework, translating knowledge into ML and software specifications, and visualizing business 
workflows and decision rules. The framework's efficacy is demonstrated in a warehouse setting, focusing on 
procurement. Experimental results reveal the successful extraction of diverse expert knowledge. 
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2. RELATED WORK 


2.1. Human Involvement in Machine Learning Workflow 


In recent years, there has been a growing interest in human involvement in the machine-learning workflow. The 
use of human-in-the-loop techniques has been proposed to improve the performance and reliability of machine 
learning models. Several studies have investigated how human expertise can be used to improve the performance 
of machine learning models. 


In the field of data science, (Wang et al., 2021) introduced AutoDS, an automated machine learning system that 
aims to support data science projects by automating tasks such as data exploration, model training, and model 
selection. This system proposes suggestions (ML configuration, pre-process data, etc.) to the users via a web- 
based graphical interface where they can interact and make amendments. They showed that the proposed system 
improved the productivity of ML workflow while delivering better models. 


In the field of aerospace systems, (D’Angelo & Palmieri, 2020) proposed the use of genetic programming to 
extract knowledge from aerospace structural defects by providing a mathematical model of the defects, which can 
be used for recognizing other similar ones. They found that their approach was effective in building reliable 
models of the defects and can be considered a successful option for building the knowledge needed by tools for 
controlling the quality of critical aerospace systems. 


(Sundin et al., 2022) proposed a principled approach to use human-in-the-loop machine learning to help chemists 
adapt the multi-parameter optimization (MPO) scoring function to better match their goal. They proposed a 
method that uses a probabilistic model that captures the user’s idea and uncertainty about the scoring function and 
uses active learning to interact with the user. They showed the effectiveness of their approach in two simulated 
examples achieving significant improvement in less than 200 feedback queries. 


Overall, these studies demonstrate the potential of a human involved in the ML workflow to improve the 
performance and reliability of machine learning models. However, further research is needed to understand the 
best ways to incorporate human expertise into the machine-learning process, and how to effectively balance the 
trade-offs between automation and human involvement. 


2.2. Fusion-driven learning 


The fusion-driven learning consists of the fusion of knowledge experts with data collected from other sources to 
improve the performance of machine learning models. Indeed, several works have been introduced to leverage 
the strengths of both human expertise and data-driven methods to create more accurate and reliable models. The 
most representative works are discussed hereafter (Ademujimi & Prabhu, 2021; Ben Brahim et al., 2022; Hu et 
al., 2019; Huang et al., 2019; Lee et al., 2020; Seymoens et al., 2019). 


(Huang et al., 2019) propose a hybrid approach for identifying the structure of the Bayesian network (BN) for the 
threat assessment of mass protests. They demonstrate that traditional methods for discovering BN structure from 
data or experts were inadequate, and instead proposed a hybrid approach (ISM-K2) which enhanced the BN 
structure learning methods via a knowledge elicitation method called ISM (Interpretive Structural Model). 


(Ademujimi & Prabhu, 2021) introduced a method for fusion-learning of Bayesian network (BN) models for fault 
diagnostics. They proposed an approach for expert knowledge elicitation of the BN structure aided by logged 
natural language data and sensor data. They found that the resulting fused BN model improved diagnostics as it 
had a wider fault coverage than the individual BNs. 


(Hu et al., 2019) developed a methodology that combines sensor data with domain expert knowledge to improve 
energy fault detection. The proposed methodology includes an engagement process with experts in the energy 
system field to identify relevant data, an integration of domain knowledge with sensor data, an automatic selection 
of potential input data, and the use of machine learning to automatically build a data-driven fault detection model. 


(Lee et al., 2020) presented an interactive machine-learning approach to improve the assessment of rehabilitation 
exercises by integrating a data-driven model with expert knowledge. This approach uses reinforcement learning 
to identify the most salient features of the exercise motions and generates a user-specific analysis to elicit feature 
relevance from a therapist for a personalized rehabilitation assessment. This study improves the performance of 
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predicting assessment and demonstrates how machine-learning models can improve with expert knowledge for 
personalized rehabilitation assessment. 


Overall, these studies demonstrate the potential of fusing knowledge from experts with data collected from other 
sources to improve the performance of machine learning models. However, further studies are needed to find 
techniques and methods to easily and efficiently fuse data used to train ML models while achieving the best 
performance. 


2.3. Knowledge Elicitation Methods 


There have been several studies in the past that have aimed to improve the efficiency and effectiveness of machine 
learning models through the incorporation of expert knowledge. These studies (Afrabandpey et al., 2019; Campos 
et al., 2018; Cheung et al., 2011; Crerie et al., 2009; El-Assady et al., 2020; El-Assady et al., 2019; Mantik et al., 
2022; Mozina et al., 2018; Park et al., 2021; Yazici et al., 2022; Young et al., 2022) have proposed various methods 
for extracting and utilizing expert knowledge: active learning, process mining and decision mining, and human- 
in-the-loop approaches. 


One widely used approach is active learning, where a model is trained on a small initial labelled dataset and then 
iteratively queries the expert for labels on the most uncertain samples. (Mozina et al., 2018) propose a data-driven 
tool for the semi-automatic identification of typical approaches and errors in student solutions for a programming 
course. They used the argument-based machine learning (ABML) method, which interactively exchanges 
arguments with an expert until the model is good enough. Similarly, (El-Assady et al., 2019) present a framework 
that integrates speculative execution, allowing users to preview the potential consequences of their actions with 
the model and make more efficient decisions. 


Another approach uses process mining and decision mining to identify operational processes, viz., business rules. 
Indeed, (Campos et al., 2018) applies a decision-mining technique in an event log of a real company to discover 
tacit decisions that could be translated as business rules. In the same way, (Crerie et al., 2009) relies on process 
mining and data mining techniques to extract two sub-types of business rules: condition action assertions and 
authorization action assertions. Likewise, (Alkofahi et al., 2022) introduces a method to elicit business rules from 
real-world web applications; these rules are defined as one-to-one and one-to-many implicit dependency 
relations, thus minimizing the negative effect of substitute relations in decision-making. 


A third approach relies on the concept of human-in-the-loop, (Afrabandpey et al., 2019; El-Assady et al., 2020; 
Park et al., 2021) whereby human experts are added into machine learning pipelines, allowing them to provide 
feedback or guidance at various stages of the model development process. For instance, (Park et al., 2021) describe 
a framework called “Ziva” that guides domain experts in sharing their knowledge with data scientists for building 
natural language processing (NLP) models. (Afrabandpey et al., 2019) introduce a method to elicit expert 
knowledge about pairwise feature similarities and use sequential decision-making techniques to minimize the 
effort of the expert while improving the prediction performance on a small dataset. (El-Assady et al., 2020) has 
developed a framework allowing users to provide semantics of their knowledge, which will contribute to topic 
model refinement. 


Finally, (Yazici et al., 2022) performs knowledge prioritization after the elicitation from domain experts. The 
authors use knowledge elicitation and feature selection techniques to identify the most prevalent tacit knowledge 
variables, which are then prioritized using machine learning methods and the fuzzy Analytic Hierarchy Process 
(AHP). 


Overall, various studies have proposed methods that allow the intuitive extraction of knowledge from experts and 
train optimized machine learning models. Although these methods allow knowledge elicitation, there are several 
areas and aspects that have not been (or have been poorly) considered so far. In this work, we aim to introduce a 
framework that considers the multi-aspect of concepts defining the context, their interdependence and translation 
into tailored specifications. 
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SECTION B - ADVANCED PROJECT MANAGEMENT AND CONTROL 


3. METHODOLOGY 


3.1. Multi-aspectual Knowledge Elicitation (MAKE) 


MAKE, multi-aspectual knowledge elicitation, developed by Winfield (Winfield, 2000) for planning and building 
knowledge-intensive systems, is based on Dooyeweerd's aspects (Table 1). By guiding and stimulating the 
participants to identify aspects that are important to their situation and opening up their constituents, MAKE 
begins with the most obvious aspects and gradually uncovers the relevance of each. As Winfield (Winfield, 2000) 
found, MAKE stimulated the participants to consider broader issues, lay participants were able to grasp the 


meaning of aspects and work with them during analysis, and some tacit knowledge was explicated through 
MAKE. 


(Winfield, 2000) developed two visual tools to help multi-aspectual analysis. One employs a flexible mind map 
to build up an understanding of inter-aspectual relationships. The second method employs the Christmas Tree, 
designed to provide an overall picture of areas of concern that emerge during discussions. Any significant positive 
or negative repercussion that emerges can be ‘hung on' the tree at the aspect in which it is meaningful, with the 
positive on one side and the negative on the other. As the picture develops, patterns emerge showing areas of 
significant benefit or problems, which can be clarified and tackled during the design and development process. 


3.2. Proposed Framework 


The framework in (Fig. 1) relies on the 15 aspects of Dooyeweerd (Basden, 2011) (Table 1) used in MAKE 
(Winfield, 2000) to elicit knowledge from domain experts through interactions, which allows an understanding of 
what is meaningful to them. Indeed, it uncovers the elements that are often not immediately apparent but contribute 
significantly to overall technology acceptance and success while avoiding unintended consequences. Our 
proposed framework consists of five key steps. 
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Fig. 1: Overview of the proposed Multi-Aspectual Knowledge Elicitation framework. 


Table 1: fifteen aspects of Dooyeweerd (Basden, 2011) and their meaning 


Aspect Meaning 
Quantitative Discrete amount 
Spatial Continuous space 
Kinematic Movement 
Physical Energy + mass, forces 
Biotic/Organic Life functions + organisms 
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Sensitive/Psychic Sense, feeling, emotion 
Analytical Distinction, conceptualization 
Formative Achievement, construction, history, technology 
Lingual Meaning carried by symbols 
Social ‘We’: relationships, roles, convention 
Economic Frugal management of resources 
Aesthetic Harmony, play, enjoyment 
Juridical Due: responsibilities + rights 
Ethical/Attitudinal Self-giving love, generosity 
Pistic/Faith Vision, aspiration, commitment, belief 


(1) Identification of business processes: analyse the company's internal/external processes to identify multiple 
cross-functional processes, data points, systems, and non-value-added operations. This step is essential to the 
identification of key data feeds (internal/external) favourable to the collection of intelligence to guide decision- 
making. 


(2) Identification of critical success factors: gather insight from senior members and staff users of the company 
to understand the business and contextual issues. This step relies on a series of interviews where the discussion 
could turn around topics like system usefulness, job security, the impact of the technology on users and their work, 
user’s attitudes to technology, skill levels and other factors that are meaningful to users. 


(3) Visual representation of extracted concepts: take concepts from step (2) and map them against 
Dooyeweerd’s aspects (e.g., technical, social, economic, ethical, etc.) to identify any gap or missing concept that 
will require further investigations (or interviews). The visual representation of extracted concepts highlights any 
laws, axioms, data, definitions and constraints that apply to the domain of the project. 


(4) Identification of tangible and non-tangible factors, extra meta-level rules and parameters: provide a 
domain conceptualisation and presentation to an expert including different aspectual views to select the aspectual 
view(s) in which experts see their domain expertise lying. This step goes through the loop of detailed knowledge 
acquisition to identify business process workflows and decisions making scenarios. 


(5) Integration of insights into the company business processes: propose a specification and design of the 
software solution to be integrated in the company information system to improve and overcome the existing 
limitations or challenges. 


3.3. Knowledge Elicitation: application 


The proposed framework relies on a series of interviews with domain experts or managers who have strong 
knowledge and understand the business processes. The application of this framework to a business starts with 
“Tutorial” interviews (Winfield, 2000), where the expert is asked to prepare a talk outlining the whole domain. , 
This helps provide an orientation to a domain and the identification of relevant concepts. The interviews are 
carried out with senior managers or team leaders who can explain the daily activities to a non-expert interviewer. 
As a result of these interviews, the interviewer should come up with internal/external processes which can impact 
the company’s objectives. 


The next step of the framework aims to identify critical factors which contribute to the success of the business 
processes. To achieve this, a “Focused” interview (Winfield, 2000) is carried out between an interviewer (ML 
Engineer) and domain experts to extract more detailed knowledge. This interview consists of three parts. First, 
there is an introduction where goals are explained to encourage the expert to take part in the discussion. Secondly, 
a set of topics is carefully chosen regarding previously identified concepts. These topics guide the interviewer to 
identify what is meaningful for experts (future users). Finally, the interviewer needs to evaluate and summarise 
the elicited knowledge before the interview ends. 


The concepts, collected during the Tutorial interviews and Focused interviews, are mapped against the fifteen 
aspects of Dooyeweerd (Basden, 2011). Indeed, it consists of the analysis of each concept to determine if it can 
be defined or interpreted by these aspects. A set of keywords can be considered as references when analysing each 
concept. A keyword can represent an entity, a process, a task, or a system. As a result, the elicited knowledge can 
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be visually illustrated and structured with the following parameters: laws, axioms, data, definitions, and 
constraints. 


After the extraction of knowledge, the latter needs to be conceptualised to the domain and presented to an expert 
for validation. This step helps narrow down the knowledge and identify tangible & non-tangible factors, rules and 
constraints involved in the decision-making process. If the validation failed, the ML engineer needs to organize 
new interviews to clarify the misalignment or collect what is missing from the expert’s knowledge. 


Finally, the ML engineer relies on elicited knowledge to propose an ML design: dataset structure (features and 
observations) and the training objectives. Indeed, it helps in the creation of a multi-variant dataset that was used 
to train a time series model for stock forecasting purposes. Moreover, the collected knowledge guides the 
definitions of specifications for the software development part of the project. Indeed, it shaped the definition of 
models, database tables, workflows and wireframes (UX/UI). 


4. RESULTS & EVALUATION 
4.1. Case study 


In this study, we worked with a wholesale company specializing in food export and distribution across the North 
UK. It operates in a multi-disciplinary environment, where teams from different disciplines work together to 
achieve the company’s objectives. The company has several business processes, such as sales, procurement, 
logistics, accounting, warehouse, e-commerce, etc. that guide its daily activities and contribute to its success. 


In a warehouse context, a procurement is a business process that involves identifying and selecting suppliers, 
negotiating contracts, and managing the purchase of food items to maintain inventory levels, meet customer 
demand and optimize costs. We applied our proposed method to the procurement business process, where we 
extracted knowledge from the team members and design machine learning models. The data and knowledge 
collected from the company’s operations were used to train ML models, which were then deployed to support the 
procurement process. 


Moreover, the company owns a bespoke resource management platform that supports various operations such as 
raising and amending purchase orders, stock management, goods-ins and quality control. The trained ML models 
were integrated into this platform to support the procurement process while offering a technology acceptance by 
the staff and final users. 


4.2. Procurement: Elicited Knowledge 


(1) Identification of business processes: the company’s business model involves several processes, from 
ordering to delivery, which aims to meet the supply-demand needs. Interviews have been conducted with the 
company staff to better understand the existing processes and come up with a supply chain map (Fig. 2). 


Fig. 2: Supply Chain Map 
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(2) Identification of critical success factors: we selected the characteristics/factors that will contribute to the 
technology acceptance of the new artificial intelligence (AI) platform with regards to the procurement business 
process. 


(3) Visual representation of extracted concepts: several concepts have been identified during the interviews 
with the domain experts. From the procurement concept, we investigated and captured knowledge in terms of 
laws, axioms, data, definitions, and constraints. The following keywords have been selected as reference for our 
investigation: product, supplier, purchase manager, carrier, reference for quotation (RFQ), purchase order, 
manufacturing, delivery, return, and credit note. Table 2 illustrates the knowledge elicited from the entity 
“Product” and mapped against the 15 aspects of Dooyeweerd. 


(4) Identification of tangible and non-tangible factors, extra meta-level rules and parameters: following the 
mapping of the key business processes against the Dooyeweerd’s aspects, we identified data attributes and 
workflows required to design a ML solution. Indeed, the work sessions with experts from the procurement domain 
allow us to come up with tacit knowledge that is illustrated in an activity diagram (Fig. 3) 


(5) Integration of insights into the company business processes: the knowledge elicited from previous steps 
allowed us to gather all specifications required to properly design databases of micro-services that will be part of 
a new software architecture in the company. Moreover, the extracted knowledge allows us to create interface 
insights (wireframes) illustrating each activity of the business process. Indeed, these wireframes (Figure 4) allows 
us to quickly validate our understanding of the business processes and ensure the technology acceptance of the 
future users. 


Table 2: Multi-aspectual knowledge elicited from an entity “Product” 


Laws Axioms Data Definitions Constraints Aspects 
Weights and A product must have | Product quantity, | A product has 
Measures Act a measurable size, weight, cost | properties which 
1985 quantity price, sales price, | can take a discrete 
online price, amount: quantity, (1) Quantitative 
; i i A product should fit 
online offer size, weight, price 
. f with the warehouse 
price, collection 
: shelves 
price 
dimensions/capacity 
Food Safety A product must have 
. Product 
Act 1990 a physical presence A product has a 
. . . dimensions, PA (2) Spatial 
(Food Safety in a specific location shape, position, 
location 
Act 1990) 
Organic A product needs to be 
Products sold before the 
Regulations expiration date. 
2009 (The 
A product must be ae? 
Organic e i A product expiration 
kept in a suitable Product . 
Products . . sme: A product has a date needs to fit with the ait A 
environment with expiration date + . . . (5) Biotic/Organic 
Regulations i life function shelf life 
respect of the shelf Shelf life 
2009), Food i 
life 
Safety Act 
1990 (Food 
Safety Act 
1990) 
General Food A product must be | Product Product can be 
Law stored and | temperature, touched, smelled E . 
i w) (6) Sensitive/ Psychic 
Regulation transported under | humidity and & tasted 
conditions that | light 
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(EC) No 


maintain its sensory 


178/2002 quality 
Food Safety A product must be A product must have a 
Act 1990 analysed and set of chemical and 
pH, nutritional Product can be 
(Food Safety evaluated for its . EN physical properties (7) Analytical 
Act 1990) BEEN bd content, shelf life | distinguished 
physical properties 
Food Standards | A product must be Product Package Some product Product rebranding 
Act 1999 capable of packages are designing should fit 
undergoing designed to meet with each market (8) Formative 
processing or customer segment 
transformation expectations 
Food Safety A product must meet | Product name, Product labelling Product name & code 
Act 1990 industry standards description, code | using symbols need to follow standards 
(Food Safety and regulatory (length, symbols, (9) Lingual 
Act 1990) requirements for languages) 
labelling 
Sale of Goods A product must be Product margin Product has a Product has to be 
Act 1979 priced in a manner benefits, limited value managed with frugality 
(11) Economic 
(Food Safety that reflects its value | profitability and 
Act 1990) growth 
Food Standards | A product must be Customers Product has to A product needs to 
Act 1999 consistent with reviews, bring joy, fun, and | satisfy the customers so i 
customer feedbacks harmony to that they get values for CaA sauce 
preferences customers what they pay for 
Food Safety A product must Product reward, Product has to A product needs to be (13) Juridical 
Act 1990 comply with relevant | recompense bring justice sold in a fair ways 
laws 
Food Safety A product must be Product Product can be A product can be (14) 
Act 1990 produced and advantages, beyond the delivered earlier, Ethical/Attitudinal 
marketed with benefits imperatives discounted 
respect of ethical 
principles 
Food Safety A product must be Dietary A product follows | A product needs to be (15) Pistic/Faith 
Act 1990 produced and restrictions, acommitment and | trustworthy 
distributed with consumer trust 
respect of spiritual preferences 


and cultural beliefs 
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Fig. 3: Activity Diagram — Make a purchase order 
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Fig 4: Procurement (Purchasing) module: this wireframe shows a quick representation of the overview page. 


4.3. Evaluation of the results 


The proposed framework was evaluated in a real-life warehouse environment where knowledge have been 
extracted from procurement experts and used to build historical sales datasets. These datasets were used to train 
time series forecasting models that optimize the procurement business process. We conducted the evaluation in 
two ways: a quantitative analysis and a qualitative analysis. 


Quantitative analysis: to quantify the performance of the proposed method, we conducted a series of experiments 
which consists of training stock forecasting models using the datasets generated from historical sales data. These 
datasets were also enriched with knowledge elicited from procurement experts using our framework. These 
methods helped to identify the learning objectives and the representation of the dataset (features, observations, 
etc). We used two time series forecasting methods like ARIMA (Harvey, 1990) & TFT (Lim et al., 2020) from 
the literature to illustrate how our framework contribute in improving the performance of the models. To evaluate 
the performance of our models we used the following metric: Quantile loss (Wen et al., 2018). Table 3 shows the 
performance of the models trained on a dataset without elicited data (D1) and a dataset with elicited data (D2). 
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Table 3: Comparison of models trained on two datasets, without (D1) and with elicited features (D2), 
w.r.t a 0.5 percentile quantile loss (p50 loss) and 0.9 percentile quantile loss (p90 loss) 


Datasets Elicited Features Model p50 loss P90 loss 
D1 - ARIMA 1.9929 1.9451 
D1 - TFT 0.6138 0.4266 
D2 Yes TFT 0.5825 0.3780 


Each dataset has the following settings: 513484 time points (about 2 years of sales data), 30 days’ horizon, and 
the stock quantity as the target feature. 


Moreover, we did a comparison of models trained on datasets, which involved features extracted with knowledge 
elicitation techniques considered as baselines (ABML, IHTM, Ziva). We used the same time series-forecasting 
model (TFT) to ensure a fair comparison (Table 4). 


Table 4: Comparison models trained on datasets generated with our proposed knowledge elicitation framework 


against baselines. 


Datasets Elicited Methods Model p50 loss P90 loss 
D2 IHTM TFT 0.6827 0.4702 
D2 Ziva TFT 0.6764 0.4629 
D2 Ours TFT 0.5825 0.3780 


Qualitative analysis: In order to evaluate the technology acceptance of solutions developed using our knowledge 
elicitation framework, we selected seven participants. This group included three individuals without prior 
knowledge in procurement and four members of the procurement team. The main objective was to determine 
whether the trained models effectively optimized business processes while considering the needs of the end user. 
Each participant was tasked with creating a purchase order on the system while adhering to two key constraints: 
avoiding stock shortages and preventing overstocking. The participants were instructed to evaluate the system 
based on several criteria, including user experience (UX), user interface (UI), workflow simplicity, and knowledge 
awareness. They rated the system on a scale of 0-5 (bad to good) for each criterion. 


In terms of user experience (UX), feedback from seven participants revealed a generally positive response to the 
system for creating or modifying purchase orders. Five participants rated the experience with a score of 5 out of 
5, indicating satisfaction, while two gave a score of 4, suggesting a desire for added features like shortcuts. 
Regarding the user interface (UI), six participants praised the new design with a score of 5, although one 
participant gave a score of 3 due to colour preferences. Evaluating workflow simplicity, three participants without 
procurement expertise rated it 4 for ease of following step-by-step instructions. In contrast, four procurement team 
members rated it 5 for consistency and accuracy. In terms of knowledge awareness, four participants rated it 5 for 
facilitating decisions on quantity, delivery, pricing, and supplier selection, while three desired more empirical data 
to bolster the system's recommendations. 


5. CONCLUSION 


In this paper, we proposed a multi-aspectual knowledge elicitation framework (MAKE4ML) for optimizing 
business processes through the design of machine-learning models. Our approach involves conducting interviews 
with domain experts and parameterize machine-learning models that can reproduce the expertise of the experts 
and provide insights for decision making. We applied the proposed framework in a food warehouse company to 
optimize the procurement process, resulting in a significant improvement in the accuracy of forecasting. 


This framework allows us to extract concepts that were relevant to the business and useful to optimize the learning 
objectives of the machine learning models. Our approach can be extended to other business processes, enabling 
efficient knowledge elicitation, and contributing to the design of machine-learning models that can optimize 
operations, reduce costs, and increase efficiency. 


Furthermore, we plan to investigate the combination of multi-aspectual knowledge elicitation techniques with the 


active learning. Active learning has been shown to be effective in reducing the amount of labelled data required 
for training machine learning models. We believe that combining active learning and the multi-aspectual 
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knowledge elicitation technique MAKE4ML can lead to even more efficient and effective optimization of 
machine learning models. 


Overall, our multi-aspectual knowledge elicitation framework can be a valuable tool for optimizing business 
processes through the design of machine-learning models. By leveraging the knowledge and expertise of domain 
experts, we can develop more effective machine learning models that can lead to cost savings, improved 
efficiency, and better decision-making. We hope that this paper provides a valuable contribution to the field and 
inspires further research in this area. 
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